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for Technology in Learning. 
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The Center for Children and Technology (CCT) is a unit of Education Development Center, Inc., anonprofit 
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and equity of education throughout the United States and in more than 35 countries. Since 1981, CCT 
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is centered in three areas: research, including basic, formative, and program evaluation; design and 
development of innovative technology prototypes and products; and the implementation and operation 
of large-scale technology integration efforts. 
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SRI International is an independent, nonprofit research institute conducting client-sponsored research and 
development for government agencies, commercial businesses, foundations, and other organizations. 
The mission of the Center for Technology in Learning (CTL) is to improve learning and teaching through 
innovation and inquiry. CTL research and development activities contribute to the knowledge base 
of effective learning and teaching and embody research insights in the innovative design, use, and 
assessment of interactive learning environments. In its development, research, and evaluation work, CTL 
seeks to create tools that lead to better teaching and learning, to develop assessments and conduct 
evaluations that contribute to the evidence base about the effectiveness and conditions for success of 
technology-supported innovations, and to inform both the policy and research communities. 
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Study in Brief 


This report presents results from the Ready To Learn Prekindergarten Transmedia Mathematics Study, 
a principal part of the summative evaluation of Ready To Learn, which is a partnership between the 
US Department of Education, the Corporation for Public Broadcasting, and PBS. Researchers found 
that preschool children who experienced a PBS KIDS Transmedia Math Supplement developed 
essential early mathematics skills. The PBS KIDS Transmedia Math Supplement was centered around 
public media videos and digital games, played on a selected set of learning technologies (interactive 
whiteboards and laptop computers). The important skills measured—counting; subitizing; recognizing 
numerals; recognizing, composing, and representing shapes; and patterning—increased significantly 
for the study’s four- and five-year-old children, who were from traditionally economically disadvantaged 
communities where children are often less prepared for kindergarten than are their more socially and 
economically advantaged peers. Also important, preschool teachers who enacted the PBS KIDS 
Transmedia Math Supplement reported significant changes in their confidence and comfort with early 
mathematics concepts and teaching with technology. 
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Introduction 


There is growing recognition of the importance of early mathematics learning, and increasing awareness 


of the tremendous potential all children have to develop a broad range of quantitative thinking skills. 
Early mathematics achievement is a strong predictor of later school achievement, and this predictive 
power is even greater than the predictive power of early literacy achievement (Claessens, Duncan, & 
Engel, 2009; Duncan et al., 2007). The National Association for the Education of Young Children (NAEYC) 
and the National Council of Teachers of Mathematics (NCTM) jointly have called attention to the need for 
appropriate, challenging, and effective early childhood mathematics programs (2002; 2010). Yet most 
preschool teachers are not trained in early mathematics content, the developmental trajectory of young 
children’s acquisition of mathematics skills, or teaching strategies to promote mathematics learning 
(Ginsburg, Lee, & Boyd, 2008). 


Despite the increased call to address mathematics in the preschool years, particularly with the introduction 
of the Common Core Standards (National Governors Association Center for Best Practices & Council of 
Chief State School Officers, 2010), few early learning mathematics curricula have been developed that are 
comprehensive in their coverage of topics beyond number and operation and geometry (Clements & Sarama, 
2009). Moreover, very few have undergone evaluations, particularly with low-income populations (Clements 
& Sarama, 2007) or using randomized controlled trials to test causal relationships between a specific 
curriculum and child outcomes (Clements & Sarama, 2002). Worse, many existing methods of teaching early 
mathematics are falling short for too many children—especially those from lower income households and 
English learners—and children who fall behind in mathematics early on face long odds of ever catching up to 
their more mathematically proficient peers when it comes to high-school graduation rates, college readiness, 
and income as adults (Duncan, et. al., 2007; NAEYC & NCTM, 2010). 
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The current study, a randomized controlled trial, explores how technology and educational transmedia 
resources can enhance prekindergarten mathematics teaching and learning in preschools, especially 
those serving children who may be at risk for academic difficulties due to economic and social 
disadvantages. This research is part of the summative evaluation of Ready To Learn, a partnership 
between the US Department of Education, the Corporation for Public Broadcasting, and PBS, that 
seeks to develop engaging, high-quality educational programming and supports for two- to eight-year- 
old children living in low-income households. A core aim of the initiative is delivering early mathematics 
(and literacy) resources on new and emerging digital platforms such as tablet computers, interactive 
whiteboards, and smartphones, as well as better-established technologies such as computers, video 
displays, and gaming consoles, and to create learning experiences that leverage the unique capabilities 
of these various technology platforms for young children’s learning. As the summative evaluation team 
for Ready To Learn, Education Development Center, Inc. (EDC) and SRI International (SRI), document 
and, whenever possible, measure the impact of transmedia mathematics resources on children’s school 
readiness. 


The study sought to test whether the experience of participating in a 10-week PBS KIDS Transmedia 
Math Supplement, which curates media-rich and non-media activities into a set curricular sequence, 
would support children’s growth of target mathematics skills: counting; subitizing; recognizing numerals; 
recognizing, composing, and representing shapes; and patterning. The materials in the supplement 
were designed to provide more structured supports for teachers to successfully enact a full range 
of mathematics instruction, as well as a way to integrate this instruction into their current teaching 
strategies and practices—both those related to mathematics and in general. Because the PBS KIDS 
Transmedia Math Supplement provided onsite technical and instructional support, a developmentally 
appropriate and sequenced set of integrated activities that target early mathematics learning, and the 
digital tools required to implement them, the research team also examined whether the experience 
of enacting the curriculum supplement enhanced teachers’ beliefs about their own understanding of 
mathematics and towards technology-enhanced early mathematics instruction. 
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Research Design 


The main goal of the current study was to determine if and how a transmedia approach with PBS KIDS 
properties serving as cornerstones of a preschool mathematics curriculum supplement, helped children 
develop targeted early mathematics skills. For the purpose of this study, transmedia means the use of 
familiar characters, settings, and narrative themes or stories across different media formats, such as 
digital video, interactive online games, and interactive whiteboard applications. In pursuit of this goal, 
the Ready To Learn summative research team identified three categories of research questions. 


Research Questions 


Child Learning Outcomes 


e What is the impact of the PBS KIDS Transmedia Math Supplement and Technology & Media experience 
on young children’s mathematics learning? 


e What is the impact of the PBS KIDS Transmedia Math Supplement and Technology & Media experience 
on young children’s self-regulation?! 


Teacher Outcomes 


e What is the impact of the PBS KIDS Transmedia Math Supplement and Technology & Media experience 
on teachers’ attitudes and beliefs about early mathematics education, and using technology and media 
to support mathematics learning? 


1 The main impact question for the study focuses on mathematics learning (namely, target skills), but the research team also examined 
potential impacts on self-regulation (skills that may be indirectly promoted through activities) as an exploratory question. 
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Implementation 


e To what extent do teachers in the PBS KIDS Transmedia Math Supplement group implement the 
curriculum supplement with fidelity? 


e What are the successes and barriers, if any, that teachers in the PBS KIDS Transmedia Math Supplement 
group encounter while implementing the curriculum supplement? 


Child and Teacher Sample 


The research team recruited a sample of 92 classrooms (46 in New York City and 46 in the San Francisco 
Bay Area) from preschool agencies and centers serving three- to five-year-old children primarily from 
low-income households. 


Screening and eligibility for recruitment into the study were based on the following requirements: 


e Centers that predominantly served children from low-income households (e.g., Head Start centers 
that have income eligibility requirements or public preschool programs that serve children who are 
eligible for state subsidy/free or reduced price lunch). 


¢ Classrooms where instruction occurred in English. 


e Classrooms that had at least 10 four- to five-year-olds enrolled who were proficient in English (given 
that resources and assessments were available only in English). 


e Participants were informed that there were three study conditions, as well as the process of random 
assignment to a particular condition. Based on this assignment, participants were willing to (1) use the 
PBS KIDS Transmedia Math Supplement; (2) receive training and coaching; (8) complete surveys; and 
(4) keep logs of their mathematics instruction. 


e Participants were willing to participate in some form of professional development in mathematics 
instruction and technology integration, and were willing to use new classroom technology. 
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Upon completion of recruitment, the research team randomly assigned participating classrooms to 
one of three conditions: a PBS KIDS Transmedia Math Supplement condition, a Technology & Media 
condition, or a Business as Usual (BAU) condition. 


The final sample of classrooms in the study consisted of 86 classrooms:? 


e 26 PBS KIDS Transmedia Math Supplement classrooms (12 in New York City; 14 in the San Francisco Bay Area) 
e 30 Technology & Media classrooms (15 in New York City; 15 in the San Francisco Bay Area) 


e 29 Business as Usual classrooms (13 in New York City; 16 in the San Francisco Bay Area) 


A sample of 157 teachers (84 in New York and 82 in the San Francisco Bay Area) participated in the 
study. The sample of teachers included lead teachers, teacher assistants, and aides. A total of 137 
teachers completed a survey and provided demographic and background information prior to the study. 


The teachers in the sample comprised a diverse group, reporting the following: 
e 44% Hispanic, Latino, or Spanish origin 

¢ 23% Black or African American 

e¢ 18% White 

e 12% Chinese 

e 7% Filipino 

e 2% American Indian or Alaskan Native 


e 2% Other Asian 


More than half of the teachers (64%) reported having completed a bachelor’s degree or higher (21.2% 
reported having completed a graduate or professional degree, 32.8% having completed a BA or BS); 
24.1% reported having completed some college or technical school classes; 17.5% reported having 
completed an AA, AS, or technical degree; and 3.6% reported having completed a high school degree 
or GED. On average, teachers reported approximately 13 years (SD=9.1) of teaching experience in 
general, and 11 years (SD=8.3) of preschool teaching experience. There were no significant differences 
between the PBS KIDS Transmedia Math Supplement and Business as Usual teachers or between 
Technology & Media and Business as Usual teachers. 


2 After randomization but before implementation, one classroom in the PBS KIDS Transmedia Math Supplement group withdrew 
from the study. Early during implementation, 13 classrooms on the East Coast were dropped from the study (6 PBS KIDS 
Transmedia Math Supplement, 3 Technology & Media group, and 5 BAU). The classrooms that were dropped moved some 
teachers to different classrooms, which resulted in potential cross-condition contamination. Because this happened at the start 
of the implementation period, the research team was able to recruit nine additional East Coast classrooms that were included 
in the study. This second cohort of classrooms was randomized to condition following the same procedure used for the original 
sample of classrooms. One of these nine additional classrooms dropped from the study before implementation began. 
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All children in participating classrooms were invited to participate in the study (engage in classroom 
activities). However, the research team randomly selected a sample of approximately 10 children from 
each participating classroom to participate in assessments of learning. Table 1 lists the total number 
of children in the overall sample and by condition, and provides descriptive information regarding 
children’s ages. 


Table 1. Total sample of children and descriptive statistics for age by condition 


OC} aTeliiteya N Mean Age AYP) 
Overall 966 4.55 0:33 
PBS KIDS Transmedia Math Supplement 307 4.55 0.34 
Technology & Media 321 4.55 033 
Business as Usual 338 4.56 O33 


The Study’s Three Conditions 


Researchers designed the study to include three conditions. In addition to the PBS KIDS Transmedia 
Math Supplement and Business as Usual groups, findings from the research team’s 2012 Prekindergarten 


Transmedia Math Pilot Study convinced us to include the Technology & Media group. Because no 


study classroom had reliable Internet access or the learning technologies included in the study, it was 
appropriate to include a study condition that reflects teachers’ typical experience with technology 
integration—receiving technology, and integrating it into their instructional routine, often with minimal 
training and support. (Table 3 provides an overview of the components of the three conditions and the 
next section describes the PBS KIDS Transmedia Math Supplement elements in greater detail. See 
Appendix A: Site Implementation Characteristics on page 52.) 


PBS KIDS Transmedia Math Supplement 


Teachers in the PBS KIDS Transmedia Math Supplement condition enacted the PBS KIDS Transmedia 
Math Supplementary curriculum, which included Ready To Learn videos and games available via public 
media broadcasting and online services, and also non-digital activities. They received preschool- 
specific interactive white boards and laptop computers, along with broadband Internet access required 
to connect these devices to the Internet, and on-demand technical support to ensure the technology 
functioned properly throughout the study period. Teachers also received pre-study training and ongoing 
coaching support focused on the enactment of the supplementary curriculum. 
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Technology & Media 


Teachers in the Technology & Media condition were asked to use new technologies and transmedia 
materials to target the same mathematics skills as the PBS KIDS Transmedia Supplement condition. 
Technology & Media classrooms were equipped with the same technology resources, and teachers 
received the same level of technology support as those in the PBS KIDS Transmedia Math Supplement 
condition. Technology & Media condition teachers received pre-study training on technology and target 
mathematics skills, and ongoing coaching to support technology integration for mathematics learning. 
Researchers did not provide the curriculum supplement. Instead, teachers were encouraged to use the 
new technologies and given pointers to digital resources they might use to enrich their existing lessons. 


Business as Usual 


Teachers in the Business as Usual control condition continued to provide the same learning opportunities 
as before the study began. They were not informed about the target mathematics skills, and continued 
to use their existing curriculum frameworks, curricula, and learning materials. They did not receive 
new technology to augment what might have been present before, and did not receive professional 
development or coaching during the study. 


Schedule 


Below is the 18-week schedule the research team followed during implementation of the current study in 
the spring of 2013. Table 2 outlines the installation, professional development, implementation, research, 
and data collection timelines for the three study conditions. 


Table 2. Study implementation and data collection schedule by week 


Put 2] 3s [a [sisi 7{[s{o]{oju]w{is|u] is | | 7 |e) 


Tech : Tech 
Technology installation (Jaro le}[ale lias @lelssicolelaai(=reialare)ele\vanie]ejexe)at Redistribution 
Child Data (Siaiife| INeolaale) Full Implementation of Curriculum Supplement (Salle) 
(exo) |[=Yos [foya} Pre-Assessments Ue) with Teachers and Children Post-Assessments 
Coachin crelelat—ig 
and PD g mcelicslolale] (@xefolelaliare} Close-out Visits 
BY=W/=\(e) e)aal=ial 


Ao edu! -19,@1(0 jEeane Taney (elant=lalrenicoyal Teacher 
room Data Survey (@lesx-1reli(elars Survey 
(@%e) |[=Yes fo) 4] 
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Table 3. Implementation Data Collection Activities 


PBS KIDS 
Transmedia Technology & Business as 


(@Xo} aKeliifelamO@xeyan) eXelat=Val ky Math Media eal 


Supplement 


Interactive Whiteboard, 3 Chromebook laptops, Vi vy 
Wireless Internet 


Printed 10-week PBS KIDS Transmedia Math V/A 
Supplement Guide with sequenced activities 


Printed 10-week guide with pointers to 
transmedia resources on PBS KIDS Lab and PBS 
KIDS websites 


Professional development for teachers on 
mathematics and use of technology 


~ 


Professional development for teachers on use of 
the PBS KIDS Transmedia Math Supplement 


~ 


Professional development on mathematics after a 
study implementation 


Hands-on materials to enact 10-week PBS KIDS 
Transmedia Math Supplement guide 


Website supporting guided transmedia 
experience with all videos and games in 
sequence 


Ongoing technology support 


Ongoing onsite coaching to support 
technology integration 


PS So Sea ex 
S 


Ongoing onsite coaching to support use of the 
PBS KIDS Transmedia Math Supplement 
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The PBS KIDS 
Transmedia Math Supplement 


The 10-week PBS KIDS Transmedia Math Supplement that was the focus of the study was a sequence 


of curricular activities centered around early childhood mathematics practices, digital resources and 
technical support to aid technology integration into instruction, and ongoing teacher coaching and 
just-in-time professional development supports. The transmedia supplement was designed to capture 
children’s interest and attention and to promote their mathematics learning, as well as support teachers’ 
understanding of their classroom mathematics practice. 


Specifically, the supplement targeted the following mathematics skills important to children’s success: 
¢ counting 

e subitizing 

® recognizing numerals 

© recognizing, composing, and representing shapes 


© patterning 


Curricular Activities 


The research team developed the PBS KIDS Transmedia Math Supplement from the ground up, drawing 
on existing research of appropriate early childhood mathematics instruction and sequencing (Clements 
& Sarama, 2009; Ginsburg, Greenes, & Balfanz, 2003), the team’s understanding of typical mathematics 
instruction in early childhood classrooms from the 2011 Context Study (Education Development Center 
& SRI International, 2011) and 2012 Prekindergarten Transmedia Math Pilot Study (EDC & SRI, 2012), 
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and existing research on successful integration of technology into early childhood classrooms (McManis 
& Gunnewig, 2012). A central component of the PBS KIDS Transmedia Math Supplement was the 
transmedia approach to digital resources, namely activities centered on interactive games designed 
by public media producers as part of the CPB-PBS Ready To Learn Initiative and companion videos 
that include familiar characters, narratives, and environments that are interesting and attractive to 
children. The PBS KIDS Transmedia Math Supplement materials were designed to act as a framework 
for teachers as they engaged in the 10-week study experience. 


In addition to the core digital assets, below are key features of the PBS KIDS Transmedia Math Supplement: 


e Overarching theme: Being a “mathematics detective.” Throughout the PBS KIDS Transmedia Math 
Supplement, children were encouraged to be “mathematics detectives” and sleuth out mathematics 
in a variety of videos, games, and activities. 


e Ready To Learn transmedia assets. These included mathematics-focused videos and interactive 
games available on the PBS KIDS Lab website. See Appendix B: Digital Assets Included in the PBS 
KIDS Transmedia Math Supplement on page 53. 


Builds on common preschool activity formats. The PBS KIDS Transmedia Math Supplement included 
whole-group, small-group, paired, and individual activities. These groupings of instruction were 
based on the most common arrangements in current prekindergarten classrooms, each with its own 
affordances for teaching and learning. 


e Structured experience. Researchers designed the PBS KIDS Transmedia Math Curriculum Supplement 
to be enacted over four days a week, for no more than 2.5 hours per week, over the course of 10 weeks. 


e Teacher’s Guide. Researchers provided teachers with printed guide containing the activities, 
suggested scripts, and suggestions for instruction for implementing the PBS KIDS Transmedia 
Math Supplement. The guide reviewed the preschool mathematics skills covered in the PBS KIDS 
Transmedia Math Supplement and provided the study plan from pre-assessment to ongoing 
implementation to post-assessment. The guide also contained tips for introducing technology to 
children; a review of the videos and games included in the PBS KIDS Transmedia Math Supplement; 
and the delivery, installation, and removal schedule for technology. See Appendix C: Teacher’s 
Guide Included in the PBS KIDS Transmedia Math Supplement on page 54. 


¢ Comprehensive materials. Teachers received all of the materials and digital assets they needed to 
enact the PBS KIDS Transmedia Math Supplement. Materials included both technology (interactive 
whiteboards and laptop computers) and manipulatives (Such as dry-erase boards and markers, 
pattern cards, and game pieces). See Appendix D: Non Digital Materials Included in the PBS KIDS 
Transmedia Math Supplement on page 55 and Appendix E: Books Included in the PBS KIDS 
Transmedia Math Supplement on page 57. 
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Researchers provided teachers inthe PBS KIDS Transmedia Math Supplement group with arecommended 
implementation schedule for a four-days-a-week implementation cycle as part of the Teacher’s Guide. 
The schedule gave suggestions for when to introduce and complete various study activities. Although 
teachers were free to follow the schedule, they were encouraged to work with their coach to tailor a 
schedule that fit their classroom needs while integrating all study activities each week. (An example is 
shown in Table 4). 


Table 4. Weekly sample schedule of the PBS KIDS Transmedia Math 
Supplement activities 


Monday Tuesday Wednesday Walelexeles 
: caus Mathematics Mathematics Circle Challenge Game 
Video Co-Viewing feck 4 
(25 minutes] De ective Journa ROU ine Play 
(20 minutes) (10 minutes) (25 minutes) 
Easy Game Play Guided Reading 
(10 minutes) (15 minutes) 
Computer Center Computer Center Computer Center Computer Center 
(~10 minutes per pair (~10 minutes per pair (~10 minutes per pair (~10 minutes per pair 
of children) of children) of children) of children) 
Hands-On Centers Hands-On Centers Hands-On Centers 
(~10 minutes per pair (~10 minutes per pair (~10 minutes per pair 
of children) of children) of children) 


Teachers in the Technology & Media condition received no supplementary curriculum. They were 
asked to build on their existing mathematics activities and content by integrating technology and 
media into their instruction. Researchers encouraged teachers to use the interactive whiteboard and 
laptop computers for digital content for approximately two hours each week to support mathematics 
instruction in the classroom and provided instructions for accessing online resources, but provided no 
formal requirements for content or in what manner to use the technology to support learning. Likewise, 
teachers were not given a specific set of mathematics skills to cover, but were encouraged to address the 
same skills emphasized in the PBS KIDS Transmedia Math Supplement condition, plus two additional 
skills that researchers felt PBS KIDS Lab games covered thoroughly: measurement, and documenting 
and analyzing data. 
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Teachers in this condition received a small Teacher’s Guide that provided information about: 
e The technology they received 

e How to integrate technology into early learning settings as a support for instruction 

e Internet safety precautions 


e Key early learning mathematics skills 


Technology Tools and Technical Supports 


All PBS KIDS Transmedia Math Supplement and Technology & Media classrooms received the technology tools 
and technical support required to participate in the study. In addition to the resources listed below, the research 
team also ensured classrooms were equipped with broadband wireless Internet by tapping into existing center 
resources or, when necessary, setting up new networks with wireless routers or mobile broadband modems. 


Teachers in the PBS KIDS Transmedia Math Supplement and Technology & Media conditions received: 


e One interactive whiteboard with mini-pc and projector, situated on an interactive whiteboard cart 
specially designed at a low height for use with young children (For an image of the interactive 
whiteboard setup, see Appendix F: Images of Interactive Whiteboard and Laptops Included in the 
PBS KIDS Transmedia Math Supplement on page 58) 


e A wireless keyboard with track pad mouse to control the interactive whiteboard 


e Three Google Chromebook laptop computers with three wireless mice and six headphones for use by 
children in the laptop center-time area 


e Headphone splitters to accommodate paired play at the laptops 


To support use of the PBS KIDS Transmedia Math Supplement by teachers, the research team developed 

a study-specific website available only to this group of teachers. Using an online open-source Content 
Management System (CMS), researchers collected links to the target transmedia resources, organized by 
week. (For an image of the site, see Appendix G: CMS Site that Supported the PBS KIDS Transmedia Math 
Supplement on page 60.) The website provided a “walled garden,” allowing access to the study’s online 
games and videos and eliminating distractions by removing access to non-study games and videos. (For an 
image of the “walled garden” approach, see Appendix G: CMS Site that Supported the PBS KIDS Transmedia 
Math Supplement on page 60.) By using a CMS to house the videos and games, the research team also was 
able to provide teachers direct access to these resources without additional materials such as DVDs or flash 
drives. Additionally, by streaming games from the PBS KIDS Lab site via the study-specific website, researchers 
were able to create unique user logins and track use of the various transmedia assets by classroom. 


Teachers in the Technology & Media condition gained access to PBS transmedia resources through 
public websites, such as www.pbskids.org/lab. 
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Teacher Supports and Coaching 


Prior to implementing the PBS KIDS Transmedia Math Supplement, teachers were required to participate 
in professional development sessions conducted by evaluation team members. These sessions consisted 
of one three- to four-hour Saturday session and another three-hour session during the same week, with 
instructional coaches encouraged to attend to meet their teachers and familiarize themselves with the 
training their teachers received. 


The sessions were designed to introduce teachers to: 


e the target mathematics skills contained in the PBS KIDS Transmedia Math Supplement (counting and 
subitizing; identifying numerals; recognizing, composing, and representing shapes; and patterning); 


° activities that composed the 10-week PBS KIDS Transmedia Math Supplement; 


e instructional strategies for using the PBS KIDS Transmedia Math Supplement activities to support 
children’s mathematics learning; 


e the Teacher’s Guide that detailed the 10-week PBS KIDS Transmedia Math Supplement; 


e the media and technology resources, and the principles behind how they were integrated into early 
mathematics teaching and learning activities; and 


e what teachers could expect from their instructional coach, as well as what their coach would expect 
of them. 


Technology & Media condition teachers also attended pre-study training that was similar in format and 
duration to that of the Transmedia condition. 


The purpose of these sessions was to introduce teachers to: 


e¢ mathematics skills to consider targeting in their classrooms (counting and subitizing; identifying 
numerals; recognizing, composing, and representing shapes; patterning; measurement; and 
documenting and analyzing data); 


ea suggested schedule for integrating mathematics and digital content into their 10-week study 
experience; 


e instructional strategies to help support children’s mathematics learning with and without technology 
and media; 


e a brief Teacher’s Guide in support of their integration efforts, including pointers to online resources; 
e the technology resources they were being asked to integrate into their classrooms; and 


e what they could expect from their coach, as well as what their coach would expect of them. 
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Additionally, during the 10-week study period both PBS KIDS Transmedia Math Supplement and 
Technology & Media condition teachers received on-site support from an instructional coach hired and 
trained by the research team. The purpose of coaching was to help teachers translate into classroom 
practice the principles and strategies presented in the professional development sessions and outlined 
in the guides provided to teachers in each of these conditions. The coaching component, which included 
support to teachers during planning and implementing lessons, allowed the coach to provide targeted 
feedback tailored to the specific needs of the teacher. 


The research team provided professional development opportunities for Business as Usual teachers at 
the conclusion of study activities in the form of a self-paced online professional development course 
from PBS TeacherLine. Teachers were enrolled in a course, “Encouraging Mathematical Communication 
Skills,” that aimed to help teachers develop communication skills to support children’s mathematical 
thinking and learning. 
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Study’s Grounding in Prior Research 


This section describes the base of prior research on which this study is built. 


Approach to Early Mathematics Instruction 


Findings from the research team’s 2012 Prekindergarten Transmedia Math Pilot Study and prior research 
(e.g., Balfanz, Ginsburg, & Greenes, 2003; Ginsburg, Choi, Lopez, Netley, & Chi, 1997) suggest that 
young children, including those in low-income communities, already use mathematical thinking and are 
able to develop new mathematical skills and knowledge. Additionally, findings from the 2012 Pilot Study 
confirmed that well-trained preschool teachers are able to guide mathematics experiences to support 
learning, and that children living in traditionally under-resourced communities often do not have access to 
a stimulating and challenging preschool learning environment that would support their early mathematics 
learning (Ginsburg et al., 2008). This research influenced the approach to early mathematics instruction 
that guided the development of the PBS KIDS Transmedia Math Supplement for the current study. 


Specifically, the content and sequencing of the PBS KIDS Transmedia Math Supplement drew on the 
work of Herb Ginsburg (Ginsburg et al., 2003), Julie Sarama, and Doug Clements (Sarama & Clements, 
2004). In particular, the Clements and Sarama Building Blocks curriculum (Sarama & Clements, 2004) 
provided strong examples of how to introduce and sequence mathematics skills that preschool teachers 
are comfortable addressing and with which they tend to be less experienced. Ginsburg’s Big Mathematics 
for Little Kids (Ginsburg et al., 2003) provided evidence-based examples of developmentally appropriate 
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mathematics topics, skills, and activities. Using these curricula as a guide—along with the Ready To Learn/ 
PBS KIDS transmedia properties available at the time of the PBS KIDS Transmedia Math Supplement 
creation—the targeted mathematics skills included in the curriculum supplement are: counting; subitizing; 
recognizing numerals; recognizing, composing, and representing shapes; and patterning. These skills were 
combined using a spiral curriculum model (Bruner, 1960), where teachers introduce content and skills and 
then provide repeated and increasingly sophisticated activities for children to understand, practice, refine, 
and master previously introduced material. 


The 2009 CPB-PBS Ready To Learn randomized controlled trial (Penuel et al., 2009) provided the theoretical 
grounding for the approach used in creating the PBS KIDS Transmedia Math Supplement. By evaluating 
the approach further during the 2012 Preschool Pilot Study of PBS KIDS Transmedia Mathematics Content 
(EDC & SRI, 2012), the research team confirmed that the combination of transmedia-supported and hands- 
on activities in a thoughtful sequence worked well in supporting teachers’ use of technology and transmedia 
for mathematics instruction. 


Approach to Transmedia and Technology 


In recent years, the early learning community has adopted a pragmatic stance toward using digital 
technologies to support learning at the preschool level. There is a growing recognition that digital media 
and technology may be used in educationally unique and developmentally appropriate ways by children, 
often with the wise guidance of parents and teachers (NAEYC/Fred Rogers Center, 2012). Still, while K-12 
teachers often embrace technology, preschool teachers are less experienced with, and have limited access 
to digital tools. In fact, only slightly more than half of preschool teachers report having a computer in their 
classroom, and most of these are non-touch screen and require young children to navigate external mice 
and keyboards (McManis & Gunnewig, 2012). Moreover, preschool teachers are unfamiliar with how to 
integrate technology into instruction (Davidson et al., 2009). Consequently, even available technology that 
is responsive to the ages and developmental levels of the children is less likely to be used. While use of 
these resources is less common in preschool, there is strong evidence for the effectiveness of public media 
content to support early learning. 


Past research has shown that developmentally appropriate PBS media and technology resources have particular 
strengths and affordances that make them powerful tools for teaching and learning at the preschool level. Public 
media programs have been shown to successfully recruit and sustain young children’s engagement, which in 
turn promotes learning across a range of domains, including mathematics (Fisch, 2004; Fisch & Truglio, 2001; 
Thakkar, Garrison, & Christakis, 2006, Penuel et al, 2009). Recent research also has shown that computer 
use can support increases in young children’s social, cognitive, and language skills, in addition to literacy and 
mathematics skills, and that touch screen technologies (e.g., interactive whiteboards) have the potential to foster 
greater collaboration and subsequent learning among preschoolers (McManis & Gunnewig, 2012). 
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This study builds on the work of McManis and Gunnewig (2012), who found that adult presence and 
guidance was required in order to make educational use of technology in early learning, and integrates 
researcher-selected transmedia content in a guiding framework for use by teachers and their children. 
McManis and Gunnewig note that adults play a dual role when supporting young children’s engagement 
with technology: They guide the use of the technology by children, and also promote peer-to-peer 
interaction around technology use. This study examines how to support adults in both these roles by 
providing guidance about how to integrate the use of technology and transmedia into instructional 
mathematics activities, and how to engage children in discussions using technology as a jumping-off 
point. The experiences were designed from the perspective that media and technology resources have 
particular strengths and affordances for learning, and are powerful tools for teaching, but are most 
valuable when thoughtfully sequenced to complement and enrich established routines and activities by 
teachers who are informed and provided with adequate resources and materials. 


Approach to Professional Development 
and Coaching 


Numerous studies suggest that successful implementation of new practices in the classroom requires 
professional development, support, and just-in-time guidance for teachers (Ackerman, 2008; Bowman, 
Donovan, & Burns, 2001; Chen & Chang, 2006; Poglinco & Bach, 2004). Thus, in the current study, 
the research team created a coaching model where “expert instructors” supported preschool teachers 
in their efforts throughout the course of study activities. The type of support provided was guided by 
findings from the evaluation team’s 2009 Ready To Learn literacy study (Penuel et al., 2009) as well as 
its 2012 Prekindergarten Transmedia Math Pilot Study (EDC & SRI, 2012) indicating that professional 
development, both prior to and targeted specifically to study activities, was an integral part of successful 
implementation of novel curriculum supplements. 


In the current study, the research team created a professional development and coaching model in 
which teachers in both the PBS KIDS Transmedia Math Supplement group and Technology & Media 
group received professional development prior to implementing any study activities, along with ongoing 
support for their efforts by an instructional coach. See Appendix H: Professional Development Model 
Included in the PBS KIDS Transmedia Math Supplement on page 62. 
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Results 


This section describes study results for children, teachers, and implementation, and includes 
methodology and the analytic approach for each. 


Child Outcomes 


A principal goal of this randomized controlled trial was to evaluate the efficacy of the PBS KIDS 
Transmedia Math Supplement and the Technology & Media condition by examining their impact on 
young children’s learning. 


Main findings include: 


e Children in the PBS KIDS Transmedia Math Supplement condition improved significantly in their 
understanding of the targeted early mathematics skills— counting, number recognition and subitizing, 
shapes, and patterns —compared to children in both the Business as Usual and Technology & Media 
conditions as measured by the supplement-based assessment. 


¢ Children in the Technology & Media condition did not improve on the target mathematics skills 
significantly compared to children in the Business as Usual condition. 


e There was no significant finding related to self-regulation as measured by the Head-Toes-Knees- 
Shoulders self-regulation assessment for children in any condition in the study. 
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Method 


To assess children’s mathematics outcomes, the research team administered two early mathematics 
assessments: a short version of the Research Based Early Mathematics Assessment (REMA short form) 
(Weiland et al., 2012), which served as a standardized assessment of children’s mathematics skills, and 
a supplement-based assessment (SBA) developed by the research team to be closely aligned to the 
mathematics concepts targeted in the PBS KIDS Transmedia Math Supplement and the Technology & 
Media conditions. In addition, to assess children’s self-regulation, the research team administered the 
Head-Toes- Knees-Shoulders (HTKS) measure (Ponitz et al., 2008), a recently developed and validated 
measure of young children’s behavioral self-regulation. Fuller descriptions of these three measures 
appear below. 


Trained assessors administered the three assessments to children in all three conditions prior to the 
beginning of the study and again after the completion of the study. Assessors administered the REMA 
on one day and the SBA and HSKT together on another day to avoid fatiguing the children. Each session 
lasted approximately 20 to 30 minutes. The research team randomly assigned assessment order to 
avoid unintended order effects. 


Standardized Assessment of Mathematics 


The short version of the REMA was recently developed by Weiland et al. (2012) based on the full REMA 
(Clements, Sarama, & Lui, 2008), which was designed to assess children’s mathematics learning 
in prekindergarten through second grade. The short version of the REMA measures preschool and 
kindergarten children’s early numeracy and geometry skills. The 19 items selected to be part of the 
short version of the REMA assess mathematics skills that are considered essential in preschool and 
kindergarten (NGA/CCSSO, 2010; Clements & Sarama, 2009)—recognition of number and subitizing, 
composition of number, comparison and sequencing, counting, numeral identification, arithmetic, shape 
and shape composition, and patterning (Weiland et al., 2012). Each item includes a game-like activity 
that involves the assessor reading a verbal prompt and, at times, demonstrating with manipulatives. 
Children are required to provide a verbal response, point, or engage with manipulatives. Rasch analysis 
conducted on the short version of the REMA provides evidence of high item reliability (approximately 
1.00) and high person reliability (ranging from 0.68 to 0.76) (Weiland et al., 2012). Findings also indicate 
that the assessment is sensitive to detect differences in young children’s mathematics ability levels and 
has adequate concurrent validity (correlations of 0.74 were reported with the full version of the REMA 
and the Woodcock Johnson Applied Problem subtest) and adequate discriminant validity (correlations 
of 0.64 were reported with the Peabody Picture Vocabulary Test 3rd Edition and the Letter Word 
Identification subscale of the Oral Language Scale). Finally, differential item functioning (DIF) analysis for 
gender, home language, and socioeconomic status across different samples indicated that there is no 
DIF in the short version of the REMA. 
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Supplement-Based Assessment of Mathematics 


To assess children’s understanding of the concepts and activities included in the PBS KIDS Transmedia 
Math Supplement, the research team developed a supplement-based assessment (SBA) during the 
2012 Pilot Study. Using information gathered from that study, and taking into account revisions to the 
curriculum supplement for the current study, a revised version of the SBA was used to assess children’s 
understanding of counting, number recognition and subitizing, shapes, and patterns. As in the REMA, 
items in the SBA involve game-like activities that require assessors to read a verbal prompt and children 
to provide a verbal response, point, or engage with manipulatives. The 20 items included in the SBA are 
aligned to the concepts and activities included in the PBS KIDS Transmedia Math Supplement. A team 
of early childhood researchers with experience in assessment developed and reviewed the SBA items 
and suggested revisions. Once items had been revised, the research team pilot-tested them on a small 
sample of preschool children and revised them as necessary. The evaluation team used data from the 
current study to examine item total correlations and found them to be higher than .2, with the majority 
ranging from .4 to .7. Analysis of SBA data also indicates high levels of internal consistency (Cronbach 
alphas were .90 and .89 for pre- and post-test, respectively). 


Standardized Assessment of Self-Regulation 


The Head-Toes-Knees-Shoulders (HTKS) was developed as a more complex version of the Head- 
to-Toes Task that was designed to measure children’s behavioral self-regulation. The Head-to-Toes 
Task involves asking children to follow a command (e.g., “Touch your toes”) and then asking them to 
respond with a conflicting or nonautomatic response (e.g., “When | say ‘Touch your toes,’ touch your 
head”). The HTKS starts off with these two commands (“Touch your toes” and “Touch your head”) 
and slowly becomes more complex by introducing two other commands (“Touch your shoulders” and 
“Touch your knees”). The HTKS requires children to use multiple cognitive skills, such as remembering 
rules and attending to verbal commands, and then applying them to their behavioral response. “These 
demands may be similar to those in classrooms, when children need to follow multiple instructions 
and finish one project before starting another or remember to raise their hand before participating” 
(Mathews, Ponitz, & Morrison, 2009, p. 693). Studies using the Head-to-Toes and HTKS have reported 
high interrater reliability (e.g., Connor et al., 2010; Ponitz, McClelland, Matthews, & Morrison, 2009; 
Wanless, McClelland, Acock, Chen, & Chen, 2011). Studies also have found that higher behavioral 
regulation (measured using the HTKS) is predictive of later mathematics and literacy achievement in 
early childhood (Ponitz et al., 2009; Wanless et al., 2011). 
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Data Analytic Approach 


To answer the study’s research questions, the evaluation team conducted a series of descriptive analyses 
and a series of multilevel analyses to examine young children’s learning in the PBS KIDS Transmedia 
Math Supplement and Technology & Media conditions, both relative to each other and relative to young 
children’s learning in a Business as Usual group, which implemented their existing practices. 


To determine the minimal detectable effect size (MDES) with power of .80, the research team conducted 
a statistical power analysis using methods outlined in Schochet (2008). Several important parameters 
must be derived to carry out sample size calculations in multilevel models: the intraclass correlation (ICC), 
which is the proportion of variance in outcome scores attributable to between-classroom variability; the 
number of clusters (classrooms); the number of children per classroom; and the proportion of variance 
that can be accounted for by covariates, which can significantly increase the precision of the estimates if 
covariates are correlated with the outcome (Spybrook, Raudenbush, Congdon, & Martinez, 2009). With 
the sample of 30 classrooms per condition (and 10 children assessed per classroom), and assuming 
that a = .05, ICC = .22 (based on previous studies using similar measures; Hedges & Hedberg, 2007) 
child- and classroom-level covariates explain approximately 65% of the student and 82% of the class- 
level variance in the outcome (also based on the previous studies cited above), the minimum detectable 
effect size (MDES) with power of .80 would be .19. 


Prior to conducting multilevel models to examine the impact of the PBS KIDS Transmedia Math Supplement 
and the Technology & Media conditions on young children’s learning, the research team examined attrition 
patterns and baseline equivalence. Researchers followed analysis procedures designed to align with What 
Works Clearinghouse (WWC) standards for evidence quality. (Detailed attrition and baseline equivalence 
analyses, a discussion of the anomalies in the HTKS measure and a discussion of heteroskedasticity are 
in Appendix I: Detailed Analysis on page 63.) 


The research team estimated main effects by conducting three-level multilevel models (students nested 
within classrooms and classrooms nested within centers) using the Stata software environment (version 
12) and the mixed command using full maximum likelihood estimation. The full specification of the main 
impact models is shown in Table 5. 
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Table 5. Main Treatment Impact HLM Specification 


Postest,, = B, + B,/_treatment,, + B,/_technology,, + 8, Pretest_centered,, 


+B, Class_Pretest_centered,, +e, + fr, +U, 


Where 
Posttest,, oe eee Sealed et Se taote ae Sean Te is the posttest score for student / within classroom j and center k. 
Pretest centered, eer vig view eestre neers is the grand mean-centered pretest score for student / within classroom / and center k. 
Class, Pretest centered, peepreerrecreet: grand mean centered pretest score for classroom j within center k. 
|_treatment,, rerere ri rer rer eceeroeer erred is an indicator for the treatment condition (1 = treatment, 0 = not treatment) of classroom j 
in center k. 
|_technology,, Bee eeeend Perce cere is an indicator for the technology condition (1 = technology, 0 = not technology) classroom 
jin center k. 
ID apa ree acre era or ee ee sre een a ae is the intercept and estimates the expected posttest score given a mean score on the 
pretest in the control condition. 
[Of serectacko eer aracaaee geese oecoenece cere eee is the estimate of the effect between the treatment and control conditions. 
| on coe gat ou aioe soecoes neo meet eceeuRE is the estimate of the effect between the technology and control conditions. 
ine iyo Upp vesseeseessntesseesstesestensensntenes are random error terms at the student, teacher, and center levels, respectively. 


Researchers note that the coefficients b, and b, are estimated in the score point metric of whatever 
assessment is being fit within the model. In order to report the impacts on three different assessments 
in a comparable scale, all impacts are converted to the standardized effect size metric known as 
Hedges’ g. Scatter plots of post-test versus pre-test measures and lines of best fit were examined for 
unusual distributions, outliers, and homogeneity of variance. 


In addition to examining main effect, the research team also investigated potential moderation effects 
by pre-test score and child age. In other words, researchers examined whether the treatment effect 
varied for children with higher vs. lower pre-test scores or for older vs. younger children. To evaluate 
potential moderation effects, interaction terms between the treatment indicators and the moderator 
were added and tested for statistical significance. 
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Results 
Child Outcomes 


Children who participated in the PBS KIDS Transmedia Math Supplement condition 
learned significantly more mathematics than did children in both the Business as Usual 
and Technology & Media conditions. 


Significant main effects relative to the Business as Usual condition were detected only for the PBS KIDS 
Transmedia Supplement condition. After implementation, children in the PBS KIDS Transmedia Math 
Supplement condition exhibited significant higher SBA scores than did children in the Business as Usual 
condition (1.51 points, effect size of 0.24, p < .001). This means that being in the PBS KIDS Transmedia 
Math Supplement condition would lead to a 9% increase in percentile rank for an average student in the 
Business as Usual group. A trend, or marginally significant effect, also was detected using the REMA 
data. After implementation, children in the PBS KIDS Transmedia Math Supplement condition exhibited 
higher REMA scores than did children in the Business as Usual condition (1.09 points, effect size of .15 
and p < .07)°. 


Significant main effects also were detected for the PBS KIDS Transmedia Supplement condition relative 
to the Technology & Media condition. After implementation, children in the PBS KIDS Transmedia Math 
Supplement condition exhibited significant higher SBA scores than did children in the Technology & 
Media condition (1.43 points, effect size of 0.22, p <. 001). This means that being in the PBS KIDS 
Transmedia Math Supplement condition would lead to a 9% increase in percentile rank for an average 
student in the Technology & Media group. A trend, or marginally significant effect, also was detected 
using the REMA data. After implementation, children in the PBS KIDS Transmedia Math Supplement 
condition exhibited higher REMA scores than did children in the Technology & Media condition (1.09 
points, effect size of .15 and p < .06). 


No significant effects were detected for HSKT scores. 


The effects of the Technology & Media condition relative to the Business as Usual condition were all 
small and not statistically significant. 


A summary table of the main effects model results follows (See Table 6). Tables of the full model 
specification and parameters appear in Appendix |: Detailed Analysis. 


3 Under NCES Statistical Standards, standard 5-1-5, “If the significance is between .05 and .10, and the observed differences 
are believed to be real, based on research or other evidence, but are not significant at the .05 level, possibly associated with 
small sample sizes and/or large standard errors, this may be noted” (National Center for Educational Statistics, 2003). 
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Table 6. Summary of PBS KIDS Transmedia Math Supplement impact estimates 


Impact Contrast 


Coefficient 
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ike) f 


la (=Xe fe [same | 
(Effect size) 
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Multiple 
Comparison 


SBA 


(1)PBS KIDS Transmedia Math Supplement 
vs Business as Usual 


(2) Technology & Media vs Business as Usual 


(3) PBS KIDS Transmedia Math Supplement 
vs. Technology & Media 


REMA 


(1)PBS KIDS Transmedia Math Supplement 
vs Business as Usual 


(2) Technology & Media vs Business as Usual 


(3) PBS KIDS Transmedia Math Supplement 
vs. Technology & Media 


HTKS 


(1)PBS KIDS Transmedia Math Supplement 
vs Business as Usual 


(2) Technology & Media vs Business as Usual 


3) PBS KIDS Transmedia Math Supplement 
vs. Technology & Media 
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0.00 
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O05 


<0.001 


0.789 


<0.001 


0.064 


0.996 


0.056 


0:77 


0.542 


0.524 


Test* 


significant 


significant 


*Note: thresholds for statistical significance adjusted for nine pair-wise comparisons using the Benjamini-Hochberg 


False Discover Rate procedure. 


While not statistically significant, findings indicate that the PBS KIDS Transmedia Math 
Supplement may have a greater benefit to those children who scored lowest on the 


supplement based assessment. 


One significant and one marginally significant moderation effect was found with pre-test in the SBA 
models. However, when accounting for inflated Type | error due to multiple comparisons, these effects 
are no longer significant or marginally significant. Although non-significant due to multiple comparisons, 
these effects merit further exploration. Appendix |: Detailed Analysis includes a table detailing both main 
and moderating effects as well as visual representation of moderator effects to aid interpretation. 


Preschool Teachers Can Use a PBS KIDS Transmedia Curriculum Supplement to Support Young Children’s Mathematics Learning 


Teacher Outcomes 
Method 


Teachers in all three conditions completed a teacher survey prior to the professional development session 
conducted at the beginning of the study, and then again at the end of the study. The teacher survey 
used in the current study was an iteration of the survey administered during the 2012 Prekindergarten 
Transmedia Mathematics Pilot (EDC & SRI, 2012). 


Both administrations of the surveys included questions about beliefs, attitudes, and practices around 
mathematics and technology use and integration. The Early Math Collaborative at the Erikson Institute 
developed the questions about beliefs, attitudes, and practices around mathematics (Chen & McCray, 
2012). In these questions, teachers are asked to rate a series of statements on a 10-point Likert scale 
(where 0 = strongly disagree and 10 = strongly agree). The research team developed the questions about 
technology use and integration based on literature reviews and a review of relevant surveys. The survey 
teachers completed prior to the study also included questions about their background and experience. 
The survey completed at the end of the study included questions about their experiences during and 
reflections after the study. 


Data Analytic Approach 


To examine the quantitative data collected in the survey, the research team conducted descriptive 
analysis by calculating frequencies and percentages for the binomial, categorical, and ordinal data, 
and means and standard deviations for the continuous data. For items included in both survey 
administrations, researchers conducted multilevel models that examined whether teachers’ responses 
pre- and post-administration were significantly different, and whether these differences varied by 
condition (Supplement vs. Business as Usual; Technology & Media vs. Business as Usual; Supplement 
vs. Technology & Media). To identify patterns in the qualitative responses, researchers examined all 
open-ended questions to develop potential coding categories. Open-ended responses were then coded 
to identify meaningful patterns that could inform findings. 
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Results 


More PBS KIDS Transmedia Math Supplement teachers reported an increased focus on 
numeracy and patterns, relative to Business as Usual teachers. 


There was a significantly greater increase in the proportion of teachers who reported teaching shapes 
in the PBS KIDS Transmedia Math Supplement condition compared to teachers in the Technology & 
Media condition. The proportion of teachers who reported teaching shapes increased from 84.4% 
to 96% in the PBS KIDS Transmedia Math Supplement condition, while in the Technology & Media 
condition it decreased slightly from 83% to 82% (b = 0.18, z = 2.04, p < 0.05). Likewise, the proportion 
of teachers who reported teaching documentation and analysis of data increased significantly in the 
PBS KIDS Transmedia Math Supplement condition relative to the Technology & Media condition. The 
proportion of such teachers increased from 47% to 84% in the PBS KIDS Transmedia Math Supplement 
Condition, while that in the Technology & Media condition increased from 46% to 50% (6 = 0.33, 
Z= 2.54, p < 0.05). 


PBS KIDS Transmedia Math Supplement and Technology & Media condition teachers 
used study technologies to promote mathematics learning. 


The proportion of teachers who reported using technology to teach mathematics concepts increased 
significantly more in both the PBS KIDS Transmedia Math Supplement and Technology & Media 
conditions, relative to the Business as Usual condition. This was true for every mathematics concept 
listed in the survey. In one instance, the proportion of teachers who reported using technology to teach 
numerals increased significantly more in the Technology & Media condition compared to that in the PBS 
KIDS Transmedia Math Supplement condition. The table below shows the proportions of teachers who 
reported teaching each mathematics concept using technology. 
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Table 7. Proportion of teachers who reported teaching mathematics concepts 
with technology 


PBS KIDS Transmedia 
Math Supplement 


Pre- Survey Post-Survey Pre-Survey Post-Survey Pre-Survey Posti-Survey 


Business as Usual i Ted a} alo) (oXe \Var--w UY -tel[e] 


Counting 59% 56% 33% 84% ** 51% 90% ** 
Subitizing 26% 28% 22% 54% * 16% 64% ** 
Identifying 

eee 50% 50% 28% 90% t 47% 84% ** 
Shapes 50% 4A% 24% 66% ** 40% 84% ** 
Patterning 46% 46% 33% 80% ** 49% 84% * 
Addition 30% 34% 22% 13 33% 7o%** 
Subtraction 24% 24% 17% 62% ** 27% 66% ** 
Measurement 15% 24% 20% To 27% 60% * 
Documenting: 17% 14% AY, 28%" 4% 34% * 


analyzing results 


*p<.05, **p<.001, tComparing PBS Transmedia Math Supplement to the Technology & Media Condition. 


The proportion of teachers who reported using videos, laptop computers, and interactive whiteboards 
specifically also increased significantly more in both the PBS KIDS Transmedia Math Supplement and 
Technology & Media conditions, relative to the Business as Usual condition. Conversely, the proportion 
of teachers who reported using desktop computers decreased in the PBS KIDS Transmedia Math 
Supplement, relative to the Business as Usual condition (6 = -.37, z = 2.26, p < .05). The proportion of 
teachers who reported using digital educational toys to promote math also decreased in the PBS KIDS 
Transmedia Math condition, relative to the Technology & Media condition (6 = -.08, z = -2.31, p < .05). 


PBS KIDS Transmedia Math Supplement teachers’ beliefs about their mathematics 
knowledge, and the benefits of technology for preschoolers improved after the study. 


PBS KIDS Transmedia Math Supplement teachers reported a greater increase in their understanding 
of the concepts of number/operations relative to Business as Usual teachers (p<.05). In the PBS 
KIDS Transmedia Math Supplement condition, the proportion of teachers who strongly agreed with 
the statement, “Il understand the concepts of number and operations as they apply to my students,” 
increased from 27% to 38%, while the proportion of Business as Usual condition teachers who 
strongly agreed decreased from 46% to 36% (6 = .37, z = 2.26, p < .05). Similarly, teachers in the 
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PBS KIDS Transmedia Math Supplement condition reported greater increases in their understanding 
of the concepts of geometry relative to both Business as Usual teachers and Technology & Media 
teachers. The proportion of PBS KIDS Transmedia Math Supplement teachers who strongly agreed 
with the statement, “Il understand the basic concepts of geometry that apply to my students,” increased 
from 16% to 40%, whereas it increased from 35% to only 38% for teachers in the Business as Usual 
condition (6 = .38, z = 2.26, p < .05), and from 24% to 30% in the Technology & Media condition (6 = 
35, Z= 2.02 , p < .05). See Figures 1 and 2. 


Figure 1. Agreement level with the statement, “| understand the concepts of 
number and operations as they apply to my students.” 
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Figure 2. Agreement level with the statement, “I understand the basic concepts 
of geometry that apply to my students." 
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PBS KIDS Transmedia Math Supplement teachers also reported greater positive changes in their beliefs 
about the impact of technology on children (6 = .29, z = 2.68, p < .01). The proportion of PBS KIDS 
Transmedia Math Supplement teachers who strongly agreed with the statement, “New technology 
encourages learning by connecting children to resources about topics they are curious about,” 
increased from 11% to 30% in the PBS KIDS Transmedia Math Supplement while it decreased from 
26% to 19% in the Business as Usual condition. Similarly, the proportion of teachers agreeing strongly 
with the statement, “New technology helps children acquire technology skills they will need in school 
and life,” increased from 25% to 45% in the PBS KIDS Transmedia Math Supplement condition, while it 
decreased from 42% to 38% in the Business as Usual condition (6 = .27, z = 1.99, p < .05). 


Figure 3. Agreement level with the statement, “New technology encourages 
learning by connecting children to resources about topics they are curious about.” 
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Figure 4. Agreement level with the statement, “New technology helps children 
acquire technology skills they will need in school and life.” 
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Most study teachers reported continuing with their regular math instructional practices 
during the study, and teachers in the PBS KIDS Transmedia Math Supplement group spent 
more time on mathematics instruction relative to teachers in the Business as Usual group. 


Some teachers received mathematics professional development apart from the professional development 
provided as part of the study. According to the post-survey, about 13% of the participating teachers 
participated in some other mathematics professional development during the study. The proportion of 
the teachers who participated in some other mathematics professional development was not significantly 
different between the PBS KIDS Transmedia Math Supplement and Business as Usual conditions, nor 
between the Technology & Media and Business as Usual conditions. 


On average, 89% of teachers said they continued with their regular math instructional activities during 
the study. PBS KIDS Transmedia Math Supplement teachers reported greater increases in the time 
spent on mathematics activities relative to Business as Usual teachers (6 = 3.7, z = 2.47, p < .05) and 
also Technology & Media teachers (b = 6.0, z = 3.29, p < 0.01.) PBS KIDS Transmedia Math Supplement 
teachers increased from 2 to 7 hours per week on average. Technology & Media teachers decreased 
from 5 to 4 hours per week on average. Business as Usual teachers increased from 2 to 4 hours per 
week on average. PBS KIDS Transmedia Math Supplement teachers reported a greater increase in the 
time spent on mathematics activities relative to Business as Usual teachers (6 = 3.7, z = 2.47, p < .05); 
PBS KIDS Transmedia Math Supplement teachers increased from 2 to 7 hours per week on average, 
while Business as Usual teachers increased from 2 to 4 hours per week on average. 
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Implementation Outcomes 


Methods 


The analyses of implementation data paint a picture of how digital media-supported early mathematics 
teaching and learning activities unfolded in PBS KIDS Transmedia Math Supplement and Technology 
& Media classrooms. Aside from highlighting contrasts across classrooms in all three study conditions 
with respect to mathematics instruction, they also provide a context for findings related to child and 
teacher outcomes described above. 


The evaluation team used four integrated data collection activities to document the implementation 
of mathematics teaching and learning activities and to describe the contrasts related to mathematics 
instruction and the use of technology and media across the PBS KIDS Transmedia Math Supplement, 
Technology & Media, and Business as Usual classrooms. These comprise weekly classroom logs, weekly 
coach logs, fidelity observations, and quality observations. The following sections describe the design of 
these research activities in detail. As the table below shows, classroom logs were completed by teachers 
in all three conditions, coach logs were completed by coaches supporting teachers in the Transmedia 
Math Curriculum Supplement Condition and Technology & Media conditions, and fidelity observations 
were conducted in Transmedia Math Curriculum Supplement classrooms. Quality observations were 
conducted in a small sample (N=18, N=6 per condition) of classrooms in all 3 conditions. 


Table 8. Implementation data collection activities by condition 


PBS KIDS 
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Math & Media LUST TT || 
Supplement 


Weekly Classroom Logs 
Completed by teachers Vv v v 


Weekly Coach Logs v v -- 


Fidelity observations 
Completed by coaches in all PBS KIDS v -- -- 
Transmedia Math Supplement classrooms 


Quality observations 

Completed by independent observers in 6 PBS WA WA Va 
KIDS Transmedia Math Supplement, Technology 

& Media, and Control classrooms (N = 18) 
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Weekly Classroom and Coach Logs 


All participating teachers in the PBS KIDS Transmedia Math Supplement, Technology & Media, and 
Business as Usual conditions completed weekly classroom logs to document the mathematics 
instruction occurring in study classrooms. Constructed as paper-and-pencil surveys that teachers filled 
out at the end of the instructional week, the classroom logs included items relating to the dosage of 
mathematics (the quantity of time spent on mathematics and the skills addressed). Additionally, PBS 
KIDS Transmedia Math Supplement teachers provided information about the implementation of the 
PBS KIDS Transmedia Math Supplement, including the extent to which they modified activities, while 
teachers in Technology & Media classrooms provided information about their use of technology and 
media resources. Both groups of teachers described the challenges they encountered while integrating 
media and technology and the types of support they received from coaches. 


Coach logs, which were completed by all coaches participating in the study (assigned to classrooms in 
the PBS KIDS Transmedia Math Supplement and Technology & Media conditions), gathered information 
about the frequency, length, and type of contact between coaches and teachers in these classrooms, 
the challenges that coaches observed teachers encountering while enacting curriculum supplement 
activities (PBS KIDS Transmedia Math Supplement classrooms) or integrating media and technology 
(PBS KIDS Transmedia Math Supplement and Technology & Media classrooms), and the types of 
support that coaches provided to teachers in both groups. 


Researchers conducted descriptive analysis of the quantitative items on the classroom and coach logs, 
computing frequencies or calculating means and standard deviations as appropriate. Additionally, the 
research team also examined contrasts across PBS KIDS Transmedia Math Supplement and Business 
as Usual conditions, and Technology & Media and Business as Usual conditions for the items where it 
was appropriate. Where available and appropriate, qualitative data help illustrate findings. 


Fidelity Observations 


Four observations were conducted in each PBS KIDS Transmedia Math Supplement classroom to 
document the extent to which classroom enactment of activities occurred with fidelity to the intervention 
as described in the Teacher’s Guide and professional development sessions. Instructional coaches 
assigned to PBS KIDS Transmedia Math Supplement classrooms conducted observations in the 
classrooms in which they provided coaching during Weeks 4 through 9 of the implementation period. 


Four activities were selected for fidelity observations: Video Co-viewing, Challenge Game Play, Math 
Circle Routine, and Guided Book Reading, as these were activities for which the PBS KIDS Transmedia 
Math Supplement provided substantial pedagogical guidance. For each activity, the research team 
identified dimensions of fidelity and developed items to document the relationship between classroom 
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enactment and the planned intervention. The fidelity observation instrument consisted of items that 
applied across activity types as well as items that were specific to individual activities. For example, 
fidelity observation items for all activities focused on the enactment of the “warm up” — whether teachers 
provided an introduction to the activity, reviewed the central mathematics skill(s), used real world 
examples, and offered children opportunities to contribute to the discussion—while the enactment of 
pause points was specific to Video Co-viewing and Guided Book Reading activities. 


The quantitative data collected through the fidelity observations were analyzed to identify frequencies or 
means and standard deviations, as appropriate. The fidelity observation data analysis also examined the 
extent of variation within classrooms, across the different activity types, as well as across classrooms. 
Similar to the coach and classroom logs, qualitative items helped provide examples for the patterns of 
fidelity of implementation. 


Quality Observations 


Researchers conducted quality observations in a randomly selected subset of six classrooms in each 
of the three conditions (for a total of 18 classrooms) to describe patterns of mathematics learning 
opportunities present in study classrooms and identify contrasts, if any, with respect to aspects of 
mathematics instruction across PBS KIDS Transmedia Math Supplement, Technology & Media, and 
Business as Usual conditions. Observations took place during implementation Weeks 6, 7, and 8. A 
total of three observations were conducted in each classroom by independent observers. 


The quality observation instrument comprised two sections. The Classroom Culture section included 
items that helped describe the classroom environment as a whole, while the Specific Math Activity 
(SMA) section, focused on mathematics teaching and learning, documented individual mathematics 
activities occurring during the observation period. The SMA section comprised a number of items 
focused on mathematics instructional practice (e.g., strategies used to support children’s mathematical 
understanding, use of mathematical talk, inviting and building on children’s mathematical contributions) 
that were rated on a Likert scale. The Classroom Observation of Early Mathematics— Environment 
and Teaching (Sarama & Clements, 2010) was the starting point for the development of the quality 
observation instrument. 


The research team aggregated ratings for the SMAs, using them as the basis for organizing classrooms 
in the observation sample into tertiles. Researchers then examined the classrooms in each tertile; the 
aim was to identify mathematics instructional practices teachers enacted consistently as well as the 
practices that were absent or appeared more challenging for teachers. The evaluation team also was 
interested in comparing classrooms in the three conditions and identifying contrasts when present. 
Because quality observations were conducted only in a small sample of study classrooms, these 
analyses are exploratory and may not generalize to the overall study sample. 
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Data Collection 


Implementation data were collected from participating study classrooms on an ongoing basis. Participating 
teachers and coaches completed weekly classroom and coach logs during all 10 weeks of implementation; 
three fidelity observations were conducted during Weeks 4 through 9 in all PBS KIDS Transmedia Math 
Supplement classrooms; and three quality observations were conducted in a sample of 18 participating 
classrooms (6 each from the PBS KIDS Transmedia Math Supplement, Technology & Media, and Business 
as Usual conditions) during Weeks 6,7, and 8 of the implementation period. 


Response rates for coach and teacher logs were higher than 95 percent, indicating that patterns of 
implementation inferred from the logs can be applied to the entire sample. Coaches and observers who 
conducted the fidelity and quality observations were trained in the use of the instruments and data collection 
procedures. For the quality observations, the training included a “guided practice” component, during 
which observers rated sample videos using the observation tool. Observers were required to demonstrate 
reliability of 70 percent or greater with an expert rater prior to data collection, and 20 percent of all quality 
observations were conducted by pairs of observers to ensure reliability of data on an ongoing basis. 


Table 9. Implementation data collection activities 


PBS KIDS Transmedia Math Technology Business as 


foul Supplement & Media Usual 
Weekly classroom logs 860 270 300 290 
Weekly coach logs 568 296 272 -- 
Fidelity observations 97 97 -- -- 
Quality observations 52 18 18 16 


Results 


Implementation of Transmedia-Supported Pedagogy 


In general, teachers implemented the PBS KIDS Transmedia Math Supplement as intended and 
addressed target mathematics skills more frequently than did teachers in the other two conditions. 


Teachers in all study conditions addressed target mathematics skills as well as other mathematics skills. Teachers 
in all three conditions commonly taught skills like counting, number identification, and patterning; however, 
teachers in the PBS KIDS Transmedia Math Supplement condition addressed target mathematics skills and 
other mathematics skills more frequently than did teachers in the Technology & Media or Business as Usual 
conditions. Furthermore, teachers in the PBS KIDS Transmedia Math Supplement condition seldom omitted 
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and rarely modified activities in the PBS KIDS Transmedia Math Supplement condition. When they modified 
activities, it was typically to break an activity into parts, shorten or extend an activity, or conduct a whole- 
class activity in small groups. Teachers typically addressed the learning goals, skills, and important points of 
each activity, and in fact often addressed more skills than the Teacher’s Guide indicated. 


Teachers typically implemented the distinctive features of the curriculum supplement—the warm-up 
and wrap-up, the video and book reading pause points, and the instructional strategies emphasized 
in the professional development. For example, there were few observed instances when the warm-up 
or wrap-up was omitted during teachers’ enactment of activities (7.22% of observations and 7.55% 
of observations respectively). However, depth of implementation varied. In particular, children were 
not always invited to contribute their thoughts to discussions during warm-up and wrap-up activities. 
Although there were multiple occasions (43.3% of the observed activities) when the warm-up included 
full explanation, called children’s attention to a real-world example, and invited children’s contributions, 
the majority of observed activities did not offer opportunities for children to contribute to the discussion 
during the warm-up. Similarly, only 34.7% of the observed wrap-ups involved an opportunity for children 
to contribute to the discussion. 


Similarly, teachers used pause points as specified in the Teacher’s Guide 95% of the time for video co- 
viewing and 94% of the time for guided book reading, often adding additional pause points of their own. 
Observers noted that discussion during the pause points centered on the mathematics knowledge and 
skills that were highlighted in the Teacher’s Guide. Moreover, observers noted the frequency of use of 
the recommended instructional strategies, including 89% of observed activities where modeling was 
used, 85% where teachers thought aloud, 73% where teachers encouraged children to think aloud, 
80% where teachers re-voiced student ideas, and 58% where teachers encouraged children to provide 
peer feedback. 


Table 10. Enactment of distinctive features of the PBS KIDS Transmedia Math 
Supplement* 


WA Te [=To) Challenge Guided Book : 
eer * Math Circle 
Co-viewing Game Play {Toe late} 
Enclenuiiesn Neus Vreldn ie) Cs 66.67% 63.65% 59.09% 73.08% 
specified in Teacher's Guide 
Enactment of wrap-up as ee ee 
specified in Teacher's Guide ML eleawe 
Enactment of pause points as 95.00% Pm 93.90% re 


specified in Teacher's Guide 


*N = 97 Fidelity observations; ** some activities did not include a wrap-up or pause points. 
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Finally, the fidelity observation data also corroborate a pattern of limited modifications. Of the 97 fidelity 
observations conducted, coaches reported observing modifications for fewer than half the activities 
(46.4%). Of the activities for which modifications were reported, coaches indicated that teachers’ 
changes, for the most part, did not conflict with the learning goal for the activity. 


Integration of digital resources, mostly from PBS KIDS, was high for Technology & Media 
condition classrooms, and teachers tended to use games and videos mostly as an 
engaging way to practice skills rather than as a way to introduce new skills or ideas. 


Analyses of classroom logs indicate that teachers in the Technology & Media condition integrated 
transmedia and technology into mathematics instruction with high frequency. During the 10-week 
implementation period, the majority of teachers reported integrating videos (73.22% of classroom 
logs) and digital games (89.15% of classroom logs) into mathematics teaching and learning activities. 
Coaches who supported teachers in identifying and selecting media resources reported that teachers 
drew from a variety of sources, but that the majority of resources came from PBS KIDS (Table 11). 


Table 11. Incorporation of transmedia into mathematics instruction (Technology 
& Media classrooms) 


PBS KIDS Other Sources t 
Videos* 75.00% 57.87% 216 coach logs 
Digital games* 96.96% 19.39% 263 coach logs 


*Totals are greater than 100% as teachers are likely to have selected videos from PBS KIDS as well as other sources 
for any given week. 


According to coaches, teachers in Technology and Media classrooms incorporated media and technology 
primarily to give children an opportunity to practice skills (96.14% of coach logs), ensure that instruction 
was more engaging for children (74.04% of coach logs) and, to a lesser extent, to provide dynamic 
representations of content (51.93% of coach logs). Also, teachers in PBS KIDS Transmedia Math 
Supplement classrooms reported using videos and computer games to a greater extent than teachers 
in Technology & Media classrooms (Figure 5), and the contrasts between conditions are significant. 
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Figure 5. Media use for mathematics instruction (Source: Classroom Logs) 
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Children’s Exposure to Mathematics 


Teachers in all study conditions addressed target math skills and other math skills, 
although PBS KIDS Transmedia Math Supplement teachers addressed these skills more 
frequently than did teachers in the other two conditions. 


In general, the target skills emphasized by the PBS KIDS Transmedia Math Supplement—counting and 
identifying numerals; subitizing; patterning; and recognizing, composing, and representing shapes— 
were addressed more frequently in PBS KIDS Transmedia Math Supplement classrooms than were other 
skills. Perhaps not surprisingly, these classrooms also addressed target skills with greater frequency 
than did Technology & Media or Business as Usual classrooms. In general, Technology & Media 
classrooms appeared to address target math skills less frequently than either PBS KIDS Transmedia 
Math Supplement or Business as Usual classrooms. Figure 6 shows the distribution of math skills 
addressed across classrooms in the three conditions. 
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Figure 6. Frequency of math skills by condition (Source: Classroom Logs) 
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Teachers in the PBS KIDS Transmedia Math Supplement condition and Technology & Media conditions 
reported spending more time on mathematics instruction per week than did teachers in the Business as 
Usual classrooms. Figure 7 illustrates the contrast. A greater number of teachers in PBS KIDS Transmedia 
Math Supplement and Technology & Media classrooms spent 4-6 hours per week on mathematics 
teaching and learning activities than did teachers in the Business as Usual condition. 
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Figure 7. Percernt of classrooms devoting 4-6 hours per week to 
mathematics instruction (Source: Classroom Logs) 
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Teachers’ Experiences Enacting Transmedia-Supported Pedagogy 


Teachers had challenges using digital resources and fitting activities into the daily 
schedule, which led to coaches offering frequent support. Notably, teachers in the 
Technology & Media condition received more on-site coach support than did teachers 
in the PBS KIDS Transmedia Math Supplement condition. 


PBS KIDS Transmedia Math Supplement teachers reported encountering an implementation challenge 
almost half the time (49% of classroom logs), and coaches reported encountering an implementation 
challenge just over half of the time (58% of coach logs). Of the challenges reported, using media and 
technology and fitting the activities into the week’s schedule were most common. To address these 
challenges, coaches assisted both PBS KIDS Transmedia Math Supplement and Technology & Media 
teachers in a variety of ways: with mathematics knowledge and skills, by modeling and co-leading 
activities, by observing instruction and providing feedback, and by assisting with materials preparation 
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for media-rich as well as hands-on activities. Although coaches provided substantial assistance to 
teachers in both conditions, teachers in the Technology & Media condition reported receiving more 
support (94% of classroom logs) than did PBS KIDS Transmedia Math Supplement teachers (74% of 
classroom logs). Teachers in the Technology & Media condition also reported an additional challenge, 
namely that of identifying and matching media to learning goals (19.32% of classroom logs). This 
finding is corroborated by the analysis of coach logs, which indicate differences between PBS KIDS 
Transmedia Math Supplement and Technology & Media classrooms with respect to the types of support 
provided by coaches (Table 12). 


Table 12. Differences in coaching support between PBS KIDS Transmedia Math 
Supplement and Technology & Media Conditions (Source: Coach Logs) 


PBS KIDS Transmedia Technology & 


YJ ol-s- eo) mexote Lod al late JU) o) ole)ai 


Math Supplement I V=Ye[fo) 
Provided support with media resources** 28.52% 5532576 
Provided support with math knowledge and skills 
and/or teaching strategies** este meee 
Co-led activities with the teacher(s)** 141% 28.47% 
Observed instruction and provided feedback 57.63% 49.81% 


** Significant; p < 0.00 


In both PBS KIDS Transmedia Math Supplement and Technology & Media conditions, the support 
provided by coaches to teachers declined during the 10-week implementation period. Although support 
provided to teachers diminished in both conditions, Technology & Media teachers continued to require 
more coaching support. Moreover, support with mathematics knowledge, skills, and instructional 
strategies remained at the same level throughout the implementation period among Technology & Media 
classrooms, while it declined in PBS KIDS Transmedia Math Supplement classrooms. 
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Limitations and Constraints 


The study has the following limitations and constraints: 


e When teaching mathematics, most preschool teachers rely on a year-long curriculum framework or one 
of the recently developed year-long curricula. The use of short, intensive supplementary experiences, 
such as the one used in this study, is rare. Compared with early mathematics curricula and curriculum 
frameworks available to many preschools, the PBS KIDS Transmedia Math Supplement focuses on a 
smaller set of mathematics skills, and relies much more on repeated experiences with technology and 
media resources to support children in learning early mathematics. 


e The sample of children for this study was not representative of preschool children in the country as 
a whole. The study was conducted in urban areas, in neighborhoods where higher percentages of 
children come from low-income households. Children in the study also were more likely to be English 
learners. 


e Measurement tools available in this area are limited. There are no standardized preschool mathematics 
measures with valid subtests for particular skills (counting, patterns, subitizing, and so on). The REMA 
was the best possible standard measure available, but measures a broader set of skills than the 
curriculum supplement covers and does not have sub-skill scales. The measure has been shown to 
be valid and reliable with children from lower-income households, like the ones recruited for this study. 


e While the available validity and reliability data for the SBA assessment is sound and the development of 
the measure was conducted according to the highest research standards, the amount of psychometric 
data available on the measure is limited when compared to standardized measures. 
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e Teacher findings are based on data that were gathered through teachers’ own replies to survey 
questions. Researchers did not independently measure teacher knowledge or attitudes. 


e While implementation logs (completed by teachers and coaches) and fidelity and quality observations 
(conducted by researchers) provide some insight into the quality and quantity of opportunities to learn 
mathematics in study classrooms, observations were not conducted in all classrooms and there is no 
way to independently verify teacher reports of the frequency and focus of mathematics lessons. 


e Given the length of the intervention—10 weeks—and the time required by teachers to become 
proficient using new educational technologies in their classrooms, long-range impacts on teacher 
practice and the associated impacts on children’s skills and Knowledge are unknown. 
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Conclusion and Future Research 


This CPB-PBS Ready To Learn study provides evidence that a curricular supplement that uses a transmedia 
approach can support young children’s mathematical skills as well as teachers’ confidence and comfort 
with early mathematics concepts and teaching with technology. While teachers whose classrooms received 
an infusion of digital resources (Technology & Media condition) integrated new technology tools and media 
resources into the mathematics lessons at quite high rates, and teachers whose classrooms were unaltered 
by the study (Business as Usual condition) spent considerable time and effort supporting children in learning 
basic mathematics skills, it was only the children in the PBS KIDS Transmedia Math Supplement group who 
improved significantly in their mathematics skills as a result of their participation. That a 10-week intervention 
could support measurable mathematics learning and teachers’ professional development is noteworthy. 
Though increasing, the number of research studies and curriculum initiatives in the field of preschool 
mathematics is limited compared to other areas of preschool learning, such as early literacy (e.g., Opening 
the World of Learning [OWL]) (Wilson, Morse, & Dickinson, 2009) and social and emotional development 
(e.g., Second Step) (Diamond, Justice, Siegler, & Snyder, 2013; Frey, Hirschstein, & Guzzo, 2000). Partial- 
year curriculum supplements, like the one studied in this randomized controlled trial, are rare both in their 
existence and as a focus of research. 


While findings from this study may offer promise to those seeking to improve learning opportunities 
available to young children from traditionally under-resourced communities, the study is likely to be 
more useful when considered together with the research team’s 2009 Ready To Learn study on digital 
media and young children’s emergent literacy skills (Penuel et al., 2009). Not only did that earlier study 
also yield positive findings pointing to digital media’s effectiveness, the pair of studies shares many of 
the same explanations for why the findings are what they are: 
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¢ Ongoing professional support is crucial for teachers and, consequently, for the children in their 
classrooms. While much attention has been focused on the promising role of technology for learning, 
and on the importance of early learning, the reality in the majority of early learning settings is little or no 
reliable access to digital tools and little or no support or training for using technology. Both Ready To Learn 
summative studies emphasized the importance of teacher professional development and support as a 
central component of any effective intervention. This emphasis grew from an examination of the literature 
suggesting that educators require training to use and integrate new tools into their instructional practice, 
and also resulted from the pilot studies that preceded each of the full studies. The types of support 
that teachers received over the course of the 10-week study period, therefore, were consequential, as 
teachers received both technical as well as pedagogical guidance. Coaches in PBS KIDS Transmedia Math 
Supplement classrooms provided feedback that was targeted toward helping teachers enact technology- 
centered and media-rich mathematics lessons intentionally designed to help children acquire, practice, 
and build on specific quantitative thinking skills. The findings suggest that, when teachers are prepared 
with the knowledge and experience needed to mediate children’s learning with technology, children will be 
able to make use of the learning opportunities available through engagement with digital media. 


e A sharp curricular focus adds potency to a common approach to technology integration, which tends 
toward the general and aspirational. Both Ready To Learn studies used a Teacher’s Guide that introduced 
and provided context for the integration of digital and non-digital activities. The full complement of activities 
provided children with exposure to and practice with a specific set of skills. Researchers designed the teacher 
guides to build on existing research about developmental trajectories for young children in mathematics 
and literacy, and also tested and modified the guides prior to final implementation. The guides served two 
key purposes: First, they laid out a concrete approach for integrating digital tools into ongoing instruction 
(both in terms of structure and also content), and second, they included pre-selected and sequenced digital 
media. Conversely, without the benefit of a sequenced set of activities, Technology & Media teachers were 
responsible not only for integrating new technology platforms into their classroom instruction but also for 
undertaking the significant design work of matching resources to planned mathematics activities, identifying 
resources, and creating larger arcs of lessons to ensure children had the repeated, linked opportunities to 
learn target skills. Seen in this way, it is perhaps not surprising that these teachers and the children they 
taught did not experience the same changes in skill and attitude toward mathematics or technology as did 
PBS KIDS Transmedia Math Supplement teachers and children. 


Teachers are the ones best able to mediate young children’s media engagement, leading to their 
learning. Even with high-quality public media mathematics or literacy resources, the presence of an adult 
whose explicit goal is to support learning is required. Teachers who orchestrate children’s experiences with 
media provide critical feedback, encouragement, direction, and other guidance, and are well positioned 
to help children interact with one another in ways that support learning beyond what solo or unmediated 
experiences with digital and non-digital resources might provide. 
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e Digital transmedia can advance content-area learning for young children, and may be of 
greatest value for those children who are most in need of academic support. Given the increasing 
recognition of the importance of early learning as a foundation for later academic success, the role 
that digital tools might play in improving learning outcomes for children who are academically at- 
risk is substantial. Most promising may be the studies’ findings that when media and technology 
are integrated into preschool settings in thoughtful and deliberate ways, they have the potential to 
help close the persistent achievement gap between children living in traditionally under-resourced 
communities who composed the sample for these experiments and their more advantaged peers. In 
the 2009 literacy study, children who scored lowest on pre-tests of early literacy skills were shown 
to have the largest gains at post-test. While additional research is needed, the promising, though 
non-significant, interaction between lower pretest scores and greater gains in the current study could 
mean that children in the bottom quartile who began the study with less-developed mathematics skills 
than their classmates (as measured by study assessments) benefitted most. These children’s early 
transmedia mathematics experiences, and their teachers’ understanding of how to bring about these 
experiences, seem to be well aligned to some of the very target skills they had not yet developed. 


Recognizing teachers as the professionals they are is both a starting place and a commitment 
that must be sustained. An important component of both studies, and perhaps a reason for the 
high rate of response and low attrition, is the research team’s focus on maintaining a relationship with 
sites that is professional and respectful of the time and effort required to participate in a research 
study spanning several months. Researchers provided information to all participants about the role 
of a randomized controlled trial in the context of an education study, the importance of having a 
Business as Usual condition and what this means for the study findings, the role of professional 
development and the provision of this for all teachers, even those in the Business as Usual condition 
(to be provided after the study was over), and the provision of stipends and a recognition of the 
time teachers spent on completing paperwork, logs, and surveys throughout the study period. One 
outcome from this is that sites are willing to work with the researcher teams’ organizations on future 
studies. Another is the ongoing use of study materials in many sites, with teachers who participated in 
different study conditions using the intervention materials provided to all sites after completion of the 
study. When teachers see that a new intervention, tool, or curriculum is developed by professionals 
who are respectful of the challenges faced and the needs and demands created by a new intervention, 
teachers are more likely to make the changes necessary to embrace and implement the intervention 
rather than dismiss it as another short-lived initiative. Researchers have had the opportunity to see 
teachers continuing to use 2009 Ready To Learn study materials, and also have observed use of the 
2013 Ready To Learn study materials by non-intervention classrooms once the study was over. 
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Finally, the lack of significant differences in children’s self-regulation skills, either for better or worse, is 
important to note. Many in the early learning community have held the belief that time spent with media 
and technology resources, no matter how well designed the resources may be or how well integrated 
into established preschool routines, can have a negative effect on children’s attention and executive 
functioning. Others have pressed the case that technology and media experiences can have a beneficial 
effect on self-regulation because they recruit and sustain children’s attention, often for extended periods 
of time. This study’s null finding provides no evidence for either side in this disagreement. 


Future Research 


The study’s implementation and findings point to several directions for ongoing research in the area of 
early learning. 


e Early learning math assessments. As the field of early mathematics assessment matures, two 
necessary advances would be the creation and validation of assessments that include subscales 
for individual skills, and that these and other assessment instruments are made valid and reliable for 
children living in under-resourced communities and English learners. Access to a broader range of 
reliable assessment options for early learning would increase the likelihood that the impact and value 
of future mathematics interventions could be measured with more accuracy. 


Professional learning. The role of coaching and the provision of professional development in early 
learning environments is also an area that could benefit from greater study to determine which 
approaches to professional development are most effective for teachers serving children who are 
at-risk academically. For example, little work has addressed how long coaching must last to provide 
long-term impact on teacher practice, or the relative benefit of training sessions that take place before 
or during implementation. 


Digital media’s alignment with a range of skills. Another area where more research can provide 
greater insight into early learning are the roles that both video and digital games are uniquely suited to 
play. This work should focus on how such resources may support children’s developing understanding 
of key concepts within mathematics and other content areas—both those concepts and skills 
commonly taught in isolation as well as others important to fuller learning progressions. Research in 
this area could include both how children can be supported with still-emerging technologies and also 
how teachers can be supported to use these tools, perhaps expanding their pedagogical and content 
knowledge. 


Work in these areas will greatly enhance future research studies and will inform the design and 
development of programs that educators can use, and from which children can benefit. 
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Appendix A: Site Implementation Characteristics 


Se i =Yed a) ate) {e) Business as 
Site Implementation Characteristics icelastaat-xellomutolia o9Y 
& Media Usual 
Supplement 
Interactive Whiteboard, 3 Chromebook laptops, y vy 
Wireless Internet 
Printed 10-week Math Curriculum Supplement vy 
Guide with sequenced activities 
Printed 10-week guide with pointers to transmedia 4 
resources on PBS KIDS Lab and PBS KIDS websites 
Professional development for teachers on math G vy, 


and use of technology 
Professional development for teachers on use 


of the PBS KIDS Transmedia Math Curriculum v 

Supplement 

Professional development on math after study vy 
implementation 

Hands-on materials to enact 10-week PBS KIDS wy, 

Math Curriculum Supplement guide 

Website supporting guided transmedia experience vy 

with all videos and games in sequence 

Ongoing technology support v v 

Ongoing onsite coaching to support technology Y Y: 

integration 

Ongoing onsite coaching to support use of the PBS VY 

KIDS Math Curriculum Supplement 

Back-end data collection of browsing history and y, Us 

use of Google Analytics 

Observation focused on fidelity of PBS KIDS Math y, 

Curriculum Supplement 

Observation protocol focused on mathematics yy Y ¢ 
instruction 

Weekly logs for teachers to reflect on their y. y, yi. 
experiences 

Weekly logs for coaches to reflect on their G Y 

experiences 

Teacher survey including beliefs and attitudes wa e. yy: 
about math and technology 

Child assessments before and after study e @ u 
implementation 

Parent survey focused on math instruction and Y e. v, 


media and technology use at home 
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Appendix B: Digital Assets Included in the PBS KIDS 


Transmedia Math Supplement 


Asset We [=To) icolatiaat-xel(o Mee lnat—ey 
Sid the Science Kid Dirt on Dirt Crystals Rule 
Vegetable Patterns 
Vegetable Harvest 
Curious George Train Master Ribbit 
Shutter Monkey Count with Allie 
Rocket Ride Apple Picking 


Zero to Donuts 


Bug Catcher 


Bunny Hunt 


Flower Garden 


Blast Off 


Bubble Pop 


Meatball Launcher 


Hide and Seek 


Bunny Ride 


Fair Shares 


The Cat in the Hat Let’s Go Fly a Kite 


Huff-Puff-a-Tron 


Termite Towers 


Sketch-a-Mite 


The Great Shape Race 


Dinosaur Train 


Buddy’s Gem Hunt 
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Appendix C: Teacher’s Guide Included in the PBS KIDS 
Transmedia Math Supplement 
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Appendix D: Non Digital Materials Included in the PBS 


KIDS Transmedia Math Supplement 


Books (7 different titles) ] Book Per Title 
Math Detective Journals 20 Journals 
Classroom Number Lines (-20 — 100) ] Number Line 
Crayons 4 Boxes 


Dry Erase Boards with Markers 


4 Boards with 1 marker each 


Dry Erase Markers 


] Package 


Dry Erase Erasers 


2 Erasers 


Personal Number Line (0-30) 


6 Number Lines 


Pipe Cleaners 3 Packages 
Shape Stickers 50 Sheets 
Foam Shapes 2 Packages 


Assorted Plastic Fruit 2 Containers 
Large Foam Dominoes ] Package 
Unifix Cubes 1 Box 
Construction Paper 2 Packages 
Large Foam Dice (1 package contains 2 dice) ] Package 
Hand Pointer ] Pointer 
Mouse Paw Stickers ] Sheet 
Clear Plastic Storage Bags (Gallon) 1 Box 

Clear Plastic Storage Bags (Quart) ] Box 

Oops! Board 0-10 4 Boards 
Oops! Number Tiles 0-10 4 Sets of Tiles 
NUMBO Boards 0-5 (1 package contains 6 boards) ] Package 
NUMBO Cards 0-5 (1 package contains 6 cards) 3 Packages 
NUMBO Boards 1-20 (1 package contains 6 boards) ] Package 
NUMBO Cards 1-20 (1 package contains 20 cards) 3 Packages 
Large Shape Cards (1 package contains 9 shapes) 2 Packages 
Simple Shape Concentration Cards (1 package contains 8 cards) 4 Packages 
Fruit Pattern Cards (AB/ABB/ABC) (1 package contains 12 cards) 2 Packages 
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Shape Sticker Pattern Cards (AB/ABB/ABC) (1 package contains 12 cards) 2 Packages 
Unifix Cube Pattem Cards (AB/ABB/ABC) (1 package contains 12 cards) 2 Packages 
Foam Shape Pattern Cards (AB/ABB/ABC) (1 package contains 12 cards) 2 Packages 
Number Tree Board (1 package contains 6 boards) 2 Package 

Fruit to Number Concentration Card (1 package contains 20 cards) 4 Packages 
Number Act Out Cards (1-5) (1 package contains 5 cards) 4 Packages 
Shape Match (1 package contains 12 sticks) 4 Packages 
Shutter Monkey Cards (1 package contains 4 cards) 2 Packages 
Body Shape Cards (1 package contains 4 cards) 2 Packages 
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Appendix E: Books Included in the PBS KIDS Transmedia 


Math Supplement 


NU i ate) af 


Illustrator 


» | Spy Numbers Jean Marzollo 


Mooncake 4 Round is a 
¢ Meera Roseann Thong 
A BOGE ae Grace Lin 
Shapes 
Busy Bugs: 
A Book About Jeviinte welmic 
Bernard Adnet 
Patterns 
ey 
Color Zoo Lois Ehlert 
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INU ale) a 
Illustrator 
Ten, Nine, 
Eight Molly Bang 
Zero Kathryn Otoshi 


Trudy Harris 
Pattern Fish Anne Canevari 
Green 


Figure Fl. The interactive white board installed on an adjustable-height cart 


with storage bins for study materials 


ni 


ie 


Preschool Teachers Can Use a PBS KIDS Transmedia Curriculum Supplement to Support Young Children’s Mathematics Learning 


Figure F2. Laptops set up for use by children in a classroom 


Figure F3. A child interacting with a PBS KIDS game on an interactive 
whiteboard 
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PBS Kids - Media by Week 
Click here to view games by week 


Week Video Guided Challenge Game Play Easy Game Play 


a Lj ; rx) > “@) 


Sid the Science Kid: Sid the Science Kid: 
The Dirt on Dirt Crystals Rule 


Sid the Science Kid: 
Vegetable Harvest 


The Cat in the Hat: | The Cat in the Hat 
Let's Go Fly a Kite Huff-Puff-a-Tron 


I The tr in the Hat: | Sid the Science Kid: IM Curious George. 
* | Finola's Farm Vegetable Patterns iam Flower Garden 


Curious George: 
Rocket Ride 


The Cat in the Hat: ‘The Cat in the Hat: Curious George: 
Termite Towers Sketch-a-Mite Bug Catcher 
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Curious George: - Curious George: Curious George: 
Zeros to Donuts Fair Shares Meatball Launcher 


I=L) == 


Curious George: | Curious George: | Curious George: 
Train Master Count With Allie Apple Picking 


Curious George: | Dinosaur Train: Curious George: 
Bunny Hunt Buddy's Gem Hunt Bunny Ride 


SVs 1 —— . ae 
Curious George: Sid the Science Kid: Curious George: 
Train Master Vegetable Patterns Flower Garden 
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Appendix H: Professional Development Model Included in 


the PBS KIDS Transmedia Math Supplement 


Teachers in the PBS KIDS Transmedia Math Supplement group 
received: 


e Seven hours of professional development over a two-day period — either off-site in a large group (by 
study staff) or on-site in small groups or one-on-one settings (by coaches) — prior to enacting the first 
week of the curriculum supplement. 


Professional Development experiences covered: 


e The target math skills contained in the PBS KIDS Transmedia Math Supplement (Counting & Subitizing; 
Identifying Numerals; Recognizing, Composing, and Representing Shapes; and Patterning) 


e Activities that comprised the ten-week PBS KIDS Transmedia Math Supplement 


e Instructional strategies for using the PBS KIDS Transmedia Math Supplement activities to support 
children’s math learning 


e The Teacher’s Guide that detailed the 10-week PBS KIDS Transmedia Math Supplement 


e The media and technology resources, and the principles behind how they were integrated into early 
math teaching and learning activities 


e What they could expect from their instructional coach as well as what their coach would expect of 
them. 
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Appendix |: Detailed Analysis 


Attrition Analysis 


The WWC Standard version 3.0 (draft) includes a procedure and table of thresholds for analyzing attrition 
in cluster randomized designs. First, attrition is assessed at the cluster level (for this study, classrooms). 
Next, using the sample of classrooms whose data may be used in the impact model, the individual 
cases are analyzed for attrition. 


The results of this study’s attrition analysis appear below. Attrition among classrooms was generally low 
overall (8%, or 3 out of 86 classrooms). The highest differential attrition percentage at the classroom 
level was 7% (PBS KIDS Transmedia Math Supplement vs. Business as Usual); this is largely due to the 
loss of two classrooms from the BAU condition and no classrooms from the PBS KIDS Transmedia Math 
Supplement condition. The research team has determined that this classroom attrition was due to a fire 
in the corresponding Center; participation in the study was discontinued for this Center. 


At the student level, conditional on classrooms being in the final sample, overall attrition ranges from 
13% to 22%. The level of attrition was higher for HTKS, likely due to the fact that more children expressed 
wanting to discontinue participation in this assessment. Differential attrition among conditions was quite 
low—analyzed without reference to clustering’, the differential attrition rates of 2% or 3% would qualify 
this analysis as meeting WWC group design standards “without reservations.” 


The attrition rates of 13% to 22% do trigger a requirement under WWC standards to analyze the data 
for baseline equivalence at pre-test. The following section summarizes that analysis. 


Baseline Equivalence Analysis 


Using the set of complete cases for each measure, the research team computed descriptive statistics 
for all three pre-test measures by condition. The team also computed pooled standard deviation across 
condition, which enabled the computation of an effect size statistic for contrasts between conditions. 
The results of this contrast analysis are shown in Table 12. The baseline differences for the PBS KIDS 
Transmedia Math Supplement vs. Business As Usual contrast ranged in absolute value from .02 to .11. 
For the Technology & Media vs. Business As Usual conditions these ranged from .04 to .15. For the PBS 
KIDS Transmedia Math Supplement vs. Technology & Media conditions these ranged from .06 to .10. 


4 The WWC Draft Standards (v3) suggest that differential attrition be analyzed at both the cluster and individual level. A 
differential attrition rate at the classroom level of 7% would qualify this report as meeting WWC standards “with reservations.” 
However, these draft standards are still under review and are subject to change. The current analysis of study quality is based 
on the currently active WWC Standards (v2.1) 
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Table |1. Table of sample sizes, attrition rates, and differential attrition rates, by 
condition and outcome measure 


Classroom 

PBS KIDS 

Transmedia Math 

Supplement OF 27 0% -7% 27 PE 0% 7% 27 o7 0% -7% 
Tech & Media 30 29 3% 4% 30 29 3% -4% 30 29 3% -4% 
BAU 29 27 7% 29 Di 7% 29 Py 7% 
*Combined* 86 83 3% 86 83 3% 86 83 3% 
Student 

PBS KIDS 

Transmedia Math 

Supplement 307 239 22% -=2% 307 257 ——s«d6 -3% 307 264 14% 0% 
Tech & Media 308 246 20% -4% 308 257 17% -3% 308 270 12% -2% 
BAU S09 288 WEY 309 248 20% 309 266 14% 
*Combined* 924 718 22% 924 762 18% 924 800 13% 


Table |I2. Standardized mean baseline differences, by treatment contrast 
and outcome 


Contrast VN HTKS SBA 
PBS KIDS Transmedia Math Supplement vs. Business As Usual -0.11 0.09 0.02 
Technology & Media vs. Business As Usual -0.04 Os OZ 
PBS KIDS Transmedia Math Supplement vs. Technology & Media -0.006 -0.06 -0.10 


The WWC Standards specify that baseline equivalence difference greater than .05 standard deviations 
but less than .25 standard deviations call for explicit adjustment for these differences to be included in 
the analytic models. While two of the nine contrasts fall under the .05 threshold (and according to the 
WWC standards do not require adjustment for baseline differences), the research team consistently 
included adjustment for baseline differences in all of the analytic models. Inclusion of baseline measures 
at the student and group levels substantially boosts statistical power, and had been the research team’s 
intention from the initial design of this evaluation. 
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Two anomalies were detected that warranted further investigation: a preponderance of “0” scores on the 
HTKS assessment, and heteroskedastic residuals for both the HTKS and SBA assessments. 


HTKS Anomalies 


In examining the HTKS assessment data, the research team discovered that 57% of the pre-test 
scores and 42% of post-test scores are zero in the analytic sample. This results in a very skewed score 
distribution, and calls into question the linear relationship between post-test and pre-test measures. The 
distribution of scores and their relationship are illustrated in Figure 11. 


An analysis fitting the squared residual of the HTKS analytic model against the pre-test predictor indicates 
inflated residual variance and a non-linearity for data where the pre-test score is less than 4. Because 
the validity of the model does not critically depend upon a true linear relationship between pre-test and 
post-test, researchers ignored the non-linearity for small pre-test values. However, the research team 
does address the heteroskedastic residuals in the next section. 


Figure |1. Distribution of HTKS pre-test and post-test scores 
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Heteroskedastic Residuals 


Researchers detected heteroskedastic residual variance in models fitting the SBA and HTKS scores; 
models fitting the REMA displayed no heteroskedasticity. Failure to adjust for non-constant residual 
variance could lead to a biased estimation of both model parameters and standard errors. 


Several solutions to this problem exist. In order to preserve the interpretation of the model parameters 
as test scores, the research team has chosen not to mathematically transform the outcome measure 
into a result that, while producing homoscedastic error variance, would be essentially uninterpretable 
as an impact model. Rather, researchers take advantage of Stata’s ability to specify HLM models with 
heteroskedastic error variance. For the HTKS models, researchers specify two levels of residual variance 
(one for cases with pre-test score less than 4, one for all other cases). For the SBA models, researchers 
specify four levels of residual variance, based on the quartile of the pre-test score. 


Figure |2 displays a color-coded scatter plot of the SBA post-test scores against centered pre-test 
scores, with approximate lines of best fit? drawn in for each of the three conditions. The slope of 
the BAU line (green) is somewhat steeper than the slopes of either the PBS KIDS Transmedia Math 
Supplement line (red) or Technology & Media line (blue). The gap between the PBS KIDS Transmedia 
Math Supplement/Technology & Media lines and the BAU line grows smaller as the pre-test scores 
increase; this is a graphical interpretation of a negative interaction coefficient shown in Table I3. In other 
words, the PBS KIDS Transmedia Math Supplement impact appears greater for students with lower 
pre-test scores, and smaller for students with higher pre-test scores. 


Figure 12. Illustration of pre-test moderation in SBA impact models 


SBA 
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@ 
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je) 

-20 20 
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5 These lines are not precisely the same as the model-estimated intercepts and slopes of the centered pre-test scores, due to inclusion of a 
group-level pre-test score in the original model. These graphs, however, serve to illustrate the moderation results shown in table |3. 


Preschool Teachers Can Use a PBS KIDS Transmedia Curriculum Supplement to Support Young Children’s Mathematics Learning 


Table 13. Summary of PBS KIDS Transmedia Math Supplement moderation model 
estimates, treatment by pre-test score 


Multiple 
I exe (=) Melee Ml mel cel aat=vi=yi Coefficient Std. Error Comparison 
Test* 

HTKS 

PBS KIDS Transmedia Math Supplement vs BAU -0.25 1.444 0.864 --- 
Tech & Media vs BAU -0.91 1.469 G35 --- 
ree ia Math Supplement 0.66 1.371 0.628 = 

PBS KIDS Transmedia Math Supplement x Pretest -0.12 0.099 0.230 --- 
Tech & Media x Pretest 0.03 0.094 05725 -- 
SRR ee ee — aos woos oe = 
REMA 

PBS KIDS Transmedia Math Supplement vs BAU ei 0570 0.061 a 
Tech & Media vs BAU -0.01 0.587 O72 aa 
Se ae eae Math Supplement vs 111 0.574 0.053 ~ 

PBS KIDS Transmedia Math Supplement x Pretest 0.10 0.069 ONIST --- 
Tech & Media x Pretest 0.00 ©1072 0.979 --- 
Techini aca Wile oe oe ng wee ates 
SBA 

PBS KIDS Transmedia Math Supplement vs BAU 72 0.321 0.000 — significant 
Tech & Media vs BAU 0.40 0.334 @:237 == 
PBS KIDS Transmedia Math Supplement vs Tech 1.33 0313 0.000 — significant 
PBS KIDS Transmedia Math Supplement x Pretest -0.07 0.037 O57 sss 
Tech & Media x Pretest -0.10 0.041 0.019 --- 

PBS KIDS Transmedia Math Supplement vs Tech 0.02 0.040 0.610 _ 


& Media x Pretest 


*Note: thresholds for statistical significance of moderator coefficients adjusted for eighteen comparisons using the 
Benjamini-Hochberg False Discover Rate procedure 
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Education Development Center, Inc. | Center for Children and Technology 


The Center for Children and Technology (CCT) is a unit of Education Development Center, Inc., a 
nonprofit international research and development organization dedicated to improving the quality, 
effectiveness, and equity of education throughout the United States and in more than 35 countries. 
Since 1981, CCT has been at the forefront of creating and researching new ways to foster learning 
and to improve teaching through the development and thoughtful implementation of new educational 
technologies. CCT’s work is centered in three areas: research, including basic, formative, and program 
evaluation; design and development of innovative technology prototypes and products; and the 
implementation and operation of large-scale technology integration efforts. 
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SRI International is an independent, nonprofit research institute conducting client-sponsored 
research and development for government agencies, commercial businesses, foundations, and 
other organizations. The mission of the Center for Technology in Learning (CTL) is to improve 
learning and teaching through innovation and inquiry. CTL research and development activities 
contribute to the knowledge base of effective learning and teaching and embody research 
insights in the innovative design, use, and assessment of interactive learning environments. In 
its development, research, and evaluation work, CTL seeks to create tools that lead to better 
teaching and learning, to develop assessments and conduct evaluations that contribute to the 
evidence base about the effectiveness and conditions for success of technology-supported 
innovations, and to inform both the policy and research communities. 
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